Optimal Stopping Rules for Sequential Hypothesis Testing

Suppose that we are given sample access to an unknown distribution p over n elements and an explicit distribution q over the same n elements. We would like to reject the null hypothesis “p = q” after seeing as few samples as possible, when p 6= q, while we never want to reject the null, when p = q. Well-known results show that Θ( √ n/ 2) samples are necessary and sufficient for distinguishing whether p equals q versus p is -far from q in total variation distance. However, this requires the distinguishing radius to be fixed prior to deciding how many samples to request. Our goal is instead to design sequential hypothesis testers, i.e. online algorithms that request i.i.d. samples from p and stop as soon as they can confidently reject the hypothesis p = q, without being given a lower bound on the distance between p and q, when p 6= q. In particular, we want to minimize the number of samples requested by our tests as a function of the distance between p and q, and if p = q we want the algorithm, with high probability, to never reject the null. Our work is motivated by and addresses the practical challenge of sequential A/B testing in Statistics. We show that, when n = 2, any sequential hypothesis test must see Ω ( 1 dtv(p,q)2 log log 1 dtv(p,q) ) samples, with high (constant) probability, before it rejects p = q, where dtv(p, q) is the—unknown to the tester—total variation distance between p and q. We match the dependence of this lower bound on dtv(p, q) by proposing a sequential tester that rejects p = q from at most O ( √ n dtv(p,q)2 log log 1 dtv(p,q) ) samples with high (constant) probability. The Ω( √ n) dependence on the support size n is also known to be necessary. We similarly provide two-sample sequential hypothesis testers, when sample access is given to both p and q, and discuss applications to sequential A/B testing. 1998 ACM Subject Classification F.2.2 Computations on discrete structures, G.3 Probability and Statistics